Papers with anchor-based inference strategy
Anchor-based Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) require massive GPU memory due to their size and parameter count. |
| Approach: | They propose to use anchor-based self-attention network and anchor-basic inference strategy to compress sequence information into an anchor token, reducing the keys/values cache and enhancing inference efficiency. |
| Outcome: | The proposed model reduces the key/value cache and improves inference efficiency by 99% while maintaining similar accuracy levels. |